Private and non-private tasks in augmented reality systems

ABSTRACT

Method, system and product for performing private and non-private tasks in Augmented Reality (AR) systems. A privacy policy of a user using an AR device is obtained. A frame sensor is utilized to obtain frames for processing. An overlay display for the frame is generated. The overlay display is generated based on an execution of a non-private task on a non-trusted device and based on an execution of a private task on a trusted device. The private task and the non-private task are determined based on the privacy policy. The overlay display is displayed by an AR device.

TECHNICAL FIELD

The present disclosure relates to computer systems in general, and toAugmented Reality (AR), in particular.

BACKGROUND

Currently AR technology has not yet fulfilled the expectations. Lack ofInternet Protocols (IP) that can support the high bandwidth and lowlatency requirements as well as privacy concerns of people are holdingwide adoption.

Some AR applications may depend on changing captured images in anaugmented reality experience. For example, an AR application may changepeople's hair, transform cats images to images of dogs, etc. Another setof applications may allow users to watch remote places and feel as ifthey are standing there, e.g., by filming a remote place from deferentangles simultaneously and creating frames depending on the location andangle of a user who is watching the remote place. Streams from eachcamera may be sent to a server to compile the different streams andprovide streams reflecting different fields of view of each user.

As the consumers of such streams may be mobile and as AR applicationsmay be required to have a large bandwidth and a low latency, FourthGeneration (4G) mobile networks may not fulfill this requirement sincethey are too slow and have a latency that is not small enough. FifthGeneration (5G) mobile networks may fulfill the connectivityrequirements since they have a bandwidth of 10 Giga bits per second(Gbps) and a latency of 1 millisecond.

When the first AR glasses entered the market they drew security andprivacy concerns. People feared from being constantly filmed by the ARglasses. People that wore it were banned from several restaurants andeven got into fights.

BRIEF SUMMARY

One exemplary embodiment of the disclosed subject matter is a methodcomprising: obtaining a privacy policy of a user, wherein the user isusing an Augmented Reality (AR) device; obtaining a frame from at leastone frame sensor; generating an overlay display for the frame, whereinsaid generating is based on an execution of a non-private task on anon-trusted device and based on an execution of a private task on atrusted device, wherein the private task and the non-private task aredetermined based on the privacy policy; and displaying, by the ARdevice, the overlay display.

Optionally, the method further comprises: determining a private portionof the frame, whereby dividing the frame to the private portion and anon-private portion; wherein the execution of the non-private taskcomprises processing the non-private portion of the frame, wherebyobtaining a first overlay; wherein the execution of the private taskcomprises processing the private portion of the frame, whereby obtaininga second overlay; wherein said generating the overlay display furthercomprises: merging the first overlay and the second overlay to obtainthe overly display.

Optionally, said determining the private portion of the frame is basedon a point of fixation of the user, wherein the non-trusted device isunable to ascertain the point of fixation of the user from thenon-private portion.

Optionally, said determining the private portion of the frame is basedon an identification of Personally Identifiable Information (PII)appearing in the frame.

Optionally, the PII comprises visible facial features of a person,whereby the non-private portion excludes the visible facial features.

Optionally, the method further comprises: determining a peripheralportion of the frame based on a point of fixation of the user; whereinthe non-private task is configured to determine a high-resolutionoverlay of the frame; wherein the private task is configured to obtain alow-resolution overlay of the peripheral portion of the frame; andwherein the overlay display for the frame comprises the low-resolutionoverlay of the peripheral portion of the frame and at least a portion ofthe high-resolution overlay of the frame.

Optionally, the trusted device comprises the AR device, wherein thenon-trusted device comprises an external device, external to the ARdevice.

Optionally, the trusted device comprises a trusted server, wherein thenon-trusted device comprises a non-trusted server.

Optionally, the non-trusted device comprises the AR device.

Optionally, the execution of the private task comprises: analyzing theframe to determine a private portion of the frame and a non-privateportion of the frame; and providing the non-private portion of the frameto the non-trusted device, whereby the non-trusted device performing theexecution of the non-private task with respect to the non-privateportion of the frame.

Optionally, the execution of the private task further comprises:obtaining a first overlay from the non-trusted device, wherein the firstoverlay is generated by the execution of the non-private task; executingone or more skins on the private portion of the frame, wherebygenerating a second overlay; and merging the first overlay with thesecond overlay, whereby generating the overlay display.

Optionally, the overlay display is determined by applying skins on theframe or a portion thereof; based on the privacy policy, identifying, inthe skins, one or more private skins and one or more non-private skins;wherein the execution of the non-private task comprises applying the oneor more non-private skins, whereby obtaining a first overlay; andwherein the execution of the private task comprises applying the one ormore private skins, whereby obtaining a second overlay; wherein saidgenerating further comprises combining the first overlay and the secondoverlay, whereby generating the overlay display.

Optionally, the execution of the non-private task further comprisesapplying the one or more redundant skins, wherein the private taskcomprises ignoring output of the one or more redundant skins.

Optionally, said applying the one or more private skins is performed ona first object in the frame, wherein the execution of the non-privatetask comprises applying the one or more private skins on a second objectin the frame.

Another exemplary embodiment of the disclosed subject matter is anAugmented Reality (AR) system comprising: an AR device having an ARdisplay; a memory configured to retain a privacy policy of a user of theAR device; and a processor configured to: determine a trust score foreach execution environment of one or more execution environments; basedon the trust score of an execution environment and based on the privacypolicy, determine a set of skins to be executed by the executionenvironment; generate overlay displays corresponding frames, wherein theframes are obtained by a frame sensor, wherein the overlay displays aregenerated, at least in part, by utilizing the execution environment toapply the set of skins on the frames; and display the overlay displaysusing the AR display.

Optionally, said determining the set of skins comprises excluding one ormore skins from being executed by the execution environment due to thetrust score being below a threshold.

Optionally, said determining the set of skins comprises determining toapply a first set of skins with respect to a first execution environmenthaving a first trust score above a threshold and determining to apply asecond set of skins with respect to a second execution environmenthaving a second trust score below the threshold.

Optionally, said determining comprises for each skin of a plurality ofskins, determining an execution environment out of the one or moreexecution environments to apply the each skin, wherein said determiningthe execution environment to apply is based on the privacy policy andthe trust scores of the one or more execution environments.

Optionally, the execution environment is characterized by acommunication channel from the AR system to the execution environment,wherein the trust score of the execution environment is determined basedon a trust score of the communication channel.

Optionally, the execution environment is characterized by a controllingentity having control of the execution environment, communicationchannel from the AR system to the execution environment, wherein thetrust score of the execution environment is determined based on a trustscore of the controlling entity.

THE BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosed subject matter will be understood and appreciatedmore fully from the following detailed description taken in conjunctionwith the drawings in which corresponding or like numerals or charactersindicate corresponding or like components. Unless indicated otherwise,the drawings provide exemplary embodiments or aspects of the disclosureand do not limit the scope of the disclosure. In the drawings:

FIG. 1 shows a schematic illustration of an exemplary environment andarchitecture in which the disclosed subject matter may be utilized, inaccordance with some exemplary embodiments of the disclosed subjectmatter;

FIG. 2 shows a flowchart diagram of a method, in accordance with someexemplary embodiments of the disclosed subject matter;

FIG. 3 shows a flowchart diagram of a method, in accordance with someexemplary embodiments of the disclosed subject matter;

FIGS. 4A-4E show a flowchart diagram of a method, in accordance withsome exemplary embodiments of the disclosed subject matter; and

FIG. 5 shows an apparatus, in accordance with some exemplary embodimentsof the disclosed subject matter.

DETAILED DESCRIPTION

One technical problem dealt with by the disclosed subject matter is toprovide an augmented reality experience to a user while utilizingcomputational power of computerized devices while hiding sensitiveinformation from a non-trusted computerized device. In some exemplaryembodiments, there may be several options for processing visualinformation, for executing skins, or the like. In some exemplaryembodiments, a skin may be an application, a software, or the like. Askin may be configured to provide an AR overlay for an AR system, ARdevice, or the like, to be used to construct an AR view for the user. Insome exemplary embodiments, a skin may be configured to process aportion of a frame, an object comprised by the frame, or the like. Askin may be applied locally, applied on an edge server, applied on aremote cloud server, or the like. In some exemplary embodiments, an ARdevice, such as AR glasses, a Virtual Retinal Display (VRD), or thelike, may be owned and controlled by the user, as opposed to otherexternal devices which are not owned or controlled by the user. As aresult, the AR device may be considered as the most trusted device. TheAR device may be configured to use an edge device, a cloud server, orthe like in order to harness their additional resources, such ascomputational power and energy. As the user may not have control overthe external devices, it may be desired desirable to limit the amount ofinformation that is shared with such devices. However, it is noted thatin some cases, the trustiness property of a device may differ based onthe context. For example, an AR device that is lent to the user by thirdparties may be considered a non-trusted device. An edge device that isowned by the user, such as an edge device installed in his car or in hishome or a mobile device acting as an edge device, may be considered as atrusted device, while an edge device installed in a rental car, in ahotel room, in a shopping mall, or the like, may be considered as anon-trusted device. As another example, the trustiness property of thecloud server may differ. In some cases, the user may trust the cloudserver as it is owned and controlled by a company that he trusts, evenmore so than an edge device controlled by a third party. In other cases,the cloud server may not be trusted. In some cases, different amount ofinformation may be exposed to a same third party depending on differentgoverning laws. For example, in a territory where the governmentrespects the privacy of citizens, more information may be exposed thanin a territory where the government has control and is able to force thethird party to reveal privately-held confidential information.

In some exemplary embodiments, the sensitive data may comprise imagesthat are being captured by a frame sensor that is operationally coupledwith the AR device. The sensitive images may comprise images of faces,children, confidential documents, places such as security zones, or thelike.

Additionally or alternatively, the sensitive data may comprise the skinsthat a user is using. As an example, a married woman may use a datingskin. She may wish to hide the usage of the skin and to prevent it frombeing executed on an external device.

Additionally or alternatively, the sensitive data may comprise metadatadetermined by the AR framework or by applied skins. For example, usingfacial recognition, identities of bystanders may be ascertained. Thefacial recognition may be applied by the AR framework itself or by aspecific skin. The information regarding the identity of the bystandermay be retained in metadata within the AR framework for futureprocessing. The exposure of the metadata to non-trusted devices may belimited, such as to prevent a wife from learning the identity of herhusband's mistress.

Another technical problem dealt with by the disclosed subject matter isto utilize information about the field of view of the user, whileavoiding sharing, directly or indirectly, such information with anon-trusted device. In some exemplary embodiments, an AR device may beconfigured to monitor, e.g., using a sensor, an eye tracking device, acamera, an infrared camera, or the like, the point of fixation of theuser. The AR device may be configured to utilize field of viewinformation in preparing the AR display. For example, in order todecrease power consumption of the AR display, resolution of the ARdisplay may be reduced in the areas of a frame that are outside thepoint of fixation of the user, or are captured by the peripheral visionof the user. As another example, in some cases, objects that are not inthe point of fixation of the user, may not be processed by skins. As anexample, a skin may analyze people and provide additional informationabout that person, such as name, position, or the like. It may bedesired to only process people that the user is looking at—e.g., withinthe user's focal point. Although the eye tracking information may beuseful and required for such processing, the user may wish to guard hisprivacy and avoid sharing such information. As an example, a user may bewatching a football with her spouse. She may not wish to reveal that shewas looking at the coach the entire match. As yet another example, aperson looking at a crowd, may not wish to share with third-parties thefact that he is looking at a woman's boozem.

One technical solution is to break the operation of providing anaugmented reality experience to a user to a private task and to anon-private task. The non-private tasks may be executed by a non-trusteddevice, the private task may be executed by a trusted device. An overlaydisplay that may comprise the output of a private task or of anon-private task may be generated and presented to a user. In someexemplary embodiments, determining the private task and the non-privatetask may be based on a privacy policy. As an example, the privacy policymay comprise a restriction requiring executing a skin that is configuredto identify faces, on a trusted device. The private task may compriseapplying the skin, and such private task may be executed by a trusteddevice, such as an edge server, the AR device, or the like. Additionallyor alternatively, the non-private task may comprise executing otherskins on non-trusted devices. As yet another example, the privacy policymay comprise a restriction on portions of the frame that are conveyed tonon-trusted devices. The privacy policy may require redaction of acontent of any visible document having a predetermined marking, such asa “CONFIDENTIAL” marking thereon. The private task may identifyconfidential content and remove such content from a frame beforetransmitting the frame to be processed (e.g., by applying skins thereon)by a non-trusted device. As yet another example, the privacy policy mayindicate for a target skin, properties of objects that are to beprocessed only by trusted devices. The private task may accordinglyidentify for each object to be processed by the target skin, whether itcomplies with the properties defined in the privacy policy. If theobject is to be processed by trusted devices only, the target skin isapplied on the object by a trusted device. The target skin may beapplied on other objects using non-trusted devices.

In some exemplary embodiments, The AR framework may obtain frames fromat least one frame sensor. In some exemplary embodiments, the privatetask and the non-private task may be executed, such as with respect tothe frames. The AR framework may obtain an overlay display. In someexemplary embodiments, obtaining the overlay display may compriseobtaining a first overlay, which may be the output of the non-privatetask. Additionally, or alternatively, obtaining the overlay display maycomprise obtaining a second overlay which may be the output of theprivate task. Additionally, or alternatively, obtaining the overlaydisplay may comprise merging or otherwise combining the first overlayand the second overlay. The overlay display may be displayed to the userusing the AR device, providing the user with an AR experience.

In some exemplary embodiments, a private portion of the frame may bedetermined. The private portion of the frame may be determined based onan identification of Personally Identifiable Information (PII) appearingin the frame. The PII may be information that may be used in order totrace the user, identify the user, or the like. As an example, the PIImay comprise visible facial features of the user (e.g. when looking at amirror). Additionally, or alternatively, the PII may comprise visiblefacial features of a person. The person may be the users' siblings,images of the users' parents, images of the users' spouse, or the like.In some exemplary embodiments, the non-private portion of the frame maybe obtained by excluding the private portion of the frame from theframe. In some exemplary embodiments, obtaining the non-private portionmay comprise deleting the private portion, such as by overwriting pixelsthereof. Additionally, or alternatively, the private portion may beremoved and pixel values thereof may not be provided as part of thenon-private portion.

In some exemplary embodiments, executing the private task may compriseprocessing the private portion of the frame. As a result fromdetermining the private portion of the frame, the frame may be dividedto a private portion and a non-private portion. In some exemplaryembodiments, executing the non-private task may comprise processing thenon-private portion of the frame. In some exemplary embodiments, the ARdevice may merge the first and second overlays in order to generate anoverlay display that is displayed to the user. As an example, theprivate portion of the frame may be determined based on the field ofview of the user. The non-private portion of the frame may be the framewithout the private portion. In some exemplary embodiments, the portionin the frame that is being watched by the user, constituting the privateportion, may be processed by the private task on a trusted device suchas on the AR device, on a trusted cloud server, on a trusted edgeserver, or the like. Additionally, or alternatively, the non-privateportion of the frame, that is not being watched by the user may, may beprocessed by the non-private task on a non-trusted device.

In some exemplary embodiments, it may be desired to save power bydownscaling the overlay display in the peripheral portion of the frame.Additionally, or alternatively, it may desired to hide from anon-trusted device the non-peripheral portion of the frame, at least bynot disclosing is the non-peripheral portion of the frame. In someexemplary embodiments, based on the field of view of the user, aperipheral portion of the frame may be determined. The non-private taskmay be configured to determine a high-resolution overlay of the frame.Additionally, or alternatively, the non-private task may be configuredto determine a high-resolution overlay and a low-resolution overlay ofthe frame. Additionally, or alternatively, the private task may beconfigured to obtain a low-resolution overlay of the peripheral portionof the frame. In some exemplary embodiments, upon obtaining ahigh-resolution overlay of the frame, the AR device may downscale aportion of the overlay corresponding to the peripheral portion of theframe. Additionally, or alternatively, the AR device may use the outputof the private task, the second overlay that may be a low resolutionoverlay of the peripheral objects of the frame and merge it with thefirst overlay.

In some exemplary embodiments, a portion of the metadata may not be sentto a non-trusted device. The metadata may comprise data such as peoplethat are with the user, the location of the user, the point of fixationthe user, or the like. A private task may be considered as private dueto comprising private metadata. By not sending the metadata the privatetask may become a non-private task, allowing to utilize resources of anon-trusted device.

In some exemplary embodiments, an AR system may determine a set of skinsto be executed. A skin may require a computerized device having a trustabove a predetermined threshold to be applied thereon. In case thatthere are no computerized devices having a trust score that is above thethreshold, the AR system may exclude that skin from the set of skins.

One technical effect of utilizing the disclosed subject matter is toexploit computational resources external to the AR device while keepingthe privacy of the user. In some exemplary embodiments, the AR devicemay use a battery and it may be desired to save electrical power byexecuting some tasks on another device. Additionally, or alternatively,the AR framework may be configured to execute tasks requiring a largeamount of computational power. Hence, it may be desired to execute sometasks on another computerized device. The disclosed subject matter maybe used to determine the private task and the non-private task, todetermine another computerized device to execute the tasks thereon inorder to save battery power and execute CPU intensive tasks whilekeeping the privacy settings.

Another technical effect of utilizing the disclosed subject matter is toexploit a trusted network and a non-trusted network based on privacysettings. In some exemplary embodiments, an AR device may be able toconnect both to a private network and to a non-private network, such asa private WiFi and a public cellular network. The AR device may utilizethe private network in order to send private data to be processed by aprivate device and the non-private data to be processed by a non-trusteddevice.

Referring now to FIG. 1 showing an environment, in accordance with someexemplary embodiments of the disclosed subject matter. Environment 100may be an environment where an AR framework is implemented. The ARframework may be a framework for providing augmented reality experienceto a user, by executing skins. Skins and AR frameworks are disclosed,inter alia, in U.S. Pat. No. 9,865,088, entitled “Evaluation ofAugmented Reality Skins”, dated Jan. 9, 2018, U.S. Pat. No. 9,953,462,entitled “Augmented Reality Skin Manager”, dated Apr. 24, 2018, U.S.Pat. No. 9,990,772B2, entitled “Augmented reality skin evaluation”,dated Jun. 5, 2018, and U.S. Patent Application Publication2016/0,240,005, entitled “Subject selected augmented reality skin”,dated Aug. 18, 2016, all of which are incorporated herein by referencein their entirety without giving rise to disavowment.

Environment 100 may comprise a Person 110. Person 110 may be wearing ARGlasses 120. AR Glasses 120 may comprise a frame sensor such as Camera124 for capturing images that are processed by the AR framework. Lensesof the AR Glasses 120 may be configured to display an overlay displaythat is seamlessly interwoven with the physical world as perceived byPerson 110, such as by adding layers over perceived objects, modifyingperceived objects, removing perceived objects, or the like.

In some exemplary embodiments, AR Glasses 120 may comprise SignalingModule 122. Signaling Module 122 may be a LED providing a visual signal,a speaker, a wireless transmitter providing a signal using radiofrequency, or the like. Signaling Module 122 may be configured toprovide a signal indicative of the restriction being imposed by the ARframework. In some exemplary embodiments, other persons may perceive thesignal and be aware that a restriction is imposed by the AR framework.Additionally or alternatively, a computerized device may be configuredto perceive the signal indicating that the AR framework is imposing arestriction. In some exemplary embodiments, Signaling Module 122 mayprovide a signal that cannot be perceived by a person, such as in anaudio frequency that is not within a human hearing range, a visualindication in an infrared light invisible to the human eye, or the like.

Environment 100 may comprise a Car 180. Car 180 may be driven by aPerson 170. Car 180 may comprise a Frame Sensor 150. Frame Sensor 150may be associated with the field of view of Person 170.

Car 180 may comprise an AR Display 160 embedded within the windshield ofCar 180. An AR framework may display the overlay display using ARDisplay 160, so as to provide an augmented display of the physicalworld. In some exemplary embodiments, frames from Frame Sensor 150 maybe analyzed by the AR framework, which may generate overlay display tobe presented using AR Display 160.

In some exemplary embodiments, Skins 125 may be utilized to provide theaugmented display. Skins 125 may comprise a plurality of skins, e.g.,Skin 125 a, Skin 125 b, and Skin 125 n, or the like, each of whichconfigured to analyze the frame or portion thereof and provide anoverlay display thereon. The overlay displays may be combined, on top ofeach other, to provide the final overlay display that is presented tothe user (e.g., Person 110, Person 170) to provide augmented realityexperience. In some exemplary embodiments, the AR framework may beconfigured to analyze the frame, identify objects therein, invoke theSkins 125 or portion thereof, collect output therefrom, and provide thefinal output for display. In some exemplary embodiments, the ARframework may be configured to invoke a skin, schedule an invocation ofa skin, determine the order in which skins are invoked, determine whichexecution environment would execute each skin, or the like.

Environment 100 may comprise an Edge 130. Edge 130 may comprise aprocessor, a receiver, a transmitter, a memory, or the like. Edge 130may be a computer, a smartphone, or the like. Similarly, an Edge 130′may be an edge computer that is installed within Car 180. Additionally,or alternatively, Edge 130′ may be a smartphone of Person 170, theinfotainment system of Car 180, or the like. Edges 130, 130′ may beutilized by the AR framework to execute one or more skins. In someexemplary embodiments, Edges 130, 130′ may constitute an executionenvironment for executing skins. Edges 130, 130′ may communicate withthe AR framework, or components via a computerized network.

Environment 100 may comprise Server 140. Server 140 be a remote cloudserver and/or any other network server. In some exemplary embodiments,Server 140 may be configured to obtain information from the ARframework, execute a skin, provide the output of the skins to the ARframework, or the like. Additionally, or alternatively, Server 140 maybe utilized by the AR framework to execute one or more skins. In someexemplary embodiments, Server 140 may constitute an executionenvironment for executing skins.

Environment 100 may comprise Frame Sensor 190 that is external to ARGlasses 120 and to Car 180. Frame Sensor 190 may be associated with thefield view of Person 170 or Person 120. Frame Sensor 190 may transmitframes to Edge 130, 130′ or to Server 140. An AR framework may utilizeframes sent from Frame Sensor 190, such as to augment the display of theuser, to identify objects to be analyzed, or the like.

In some exemplary embodiments, the AR device, such as AR Glasses 120, ARDisplay 160, or the like, may be connected to an edge computer, such asEdges 130, 130′, and to cloud server, e.g., Server 140. The edgecomputer, however, may be closer to the AR device than the cloud server,such as having a reduced latency thereto, requiring reducing powerconsumption to reach it, or the like. For example, the edge computer maybe directly connected, such as using a Bluetooth connection, using WiFiconnection, using local cellular connection, or the like. Additionally,or alternatively, the edge computer may be connected to a same LocalArea Network (LAN) as the AR device, may be physically connected theretousing a cabled connection, or the like. In some exemplary embodiments,communication to Server 140 may be provided through a Wide Area Network(WAN), through the Internet, or the like. For example, Server 140 may bea cloud computing platform that is used by various AR devices that arelocated remotely therefrom and from one another.

In some exemplary embodiments, different execution environments may havedifferent resources. For example, Server 140 may have more computationalresources, memory resources, or the like than Edge 130. Additionally, oralternatively, each execution environment may have different powerresources. The AR device, such as AR Glasses 120 may have limited powersupply. Additionally, or alternatively, Edge 130, which may beimplemented by a mobile device, may also have limited power supply.Additionally, or alternatively, different execution platforms mayretain, locally, or have connectivity to, different data storages. Forexample, a database used for identifying a person using facialrecognition may be retained on Server 140. AR Glasses 120 may havelimited memory capacity and may be able to retain only a portion of suchdatabase, such as a portion comprising frequently identified persons,people who are known to be at the same location, people that are deemedVIPs for the user, or the like. As a result, it may be desired toperform some computations off-device, on Edge 130 or on Server 140. Asan example, Skin 125 a may be executed on AR Glasses 120, Skin 125 b maybe executed on Edge 130 and Skin 125 n may be executed on Server 140. Insome exemplary embodiments, each skin may receive, as input, the outputof the previous skin, in accordance with an order of execution, so as toadd a layer on top thereof. As an example, Skin 125 a may add an imageof a cat, and Skin 125 b may process the image of the cat and modify itsappearance. The AR framework may be configured to invoke execution ofeach skin, in accordance with predetermined rules and based on thecontent of the frame, to select for each skin an execution environment,to gather input for the skin and obtain the output of the skin, andgenerate the final overlay display to be displayed by the AR device.

Referring now to FIG. 2 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 210, the framework obtains a frame. The frame may be obtained bya frame sensor, such as camera, that may be operatively coupled with theframework. The frame may be obtained directly from a frame sensor thatis associated with a view of a user, such as a camera embedded in ARglasses (e.g., 124 of FIG. 1). Additionally or alternatively, the framemay be obtained from a computerized environment such as an edge device,a remote cloud server, or the like. Additionally or alternatively, theframe may be obtained from an external frame sensor, such as 190 of FIG.1, that may be utilized to capture information that may be obscured andblocked from the view of the user. In some exemplary embodiments, imagesmay be obtained from a plurality of sources and may be integratedtogether to create the frame.

On Step 220, the framework identifies objects that are in the frame. Theframework may be configured to identify objects using algorithms such asRegions Convolutional Neural Network (RCNN), Faster RCNN,Scale-Invariant Feature Transform (SIFT), template matching, edgematching, or the like. In some exemplary embodiments, each skin mayregister on one or more object type. In some exemplary embodiments, inorder to improve performance, the framework may only try to identifyobjects on which at least one skin is registered. As an example, in casethat skins are registered on objects of type “face” and “hat”, theframework may skip large objects such as a car, a house, or the like. Insome exemplary embodiments, the framework may analyze the frame andmaintain a metadata record representing the frame identifying eachobject appearing in the frame. Additionally or alternatively, themetadata may be indicative to the surrounding of the user, to peoplethat may be with the user, or the like.

On Step 230, the framework may invoke a skin. The framework may hold alist of available skins and corresponding objects and pull informationregarding the faces from a social network database. The woman may entera nudist beach. At the entrance to the nudist beach, there may be amarker signaling to the AR glasses not to perform facial recognition.The AR framework may embed that restriction in a metadata. The skin mayobtain the metadata and pause its operation until the restriction islifted. In some exemplary embodiments, the skin may unregister from theobjects in the AR framework, and register for a notification from the ARframework when the altered functionality is lifted. Upon receiving sucha notification, the skin may re-register on the objects of interest. Asa result, performance improvement is achieved as the skin is neverinvoked when it is unable to perform its intended operation due to themarker-related instructions. Additionally or alternatively, the ARframework may manage the unregistration and registration of pertinentskins, instead of the skin itself unregistering and re-registeringitself.

On Step 240, the framework obtains an overlay that is the output of theskin. In some exemplary embodiments, in case that the skin was notexecuted locally (e.g. executed on an edge computer, on a cloud server,or the like), the overlay may be obtained therefrom via a communicationchannel. In some exemplary embodiments, the skin may be executed locallyas function. The overlay may be the output of the function. Additionallyor alternatively, the skin may be executed remotely. The overlay may beprovided to the AR framework as a stream. Additionally or alternatively,the overlay may comprise a bitmap image and a bitmap mask. The bitmapimage may comprise the content of the pixels to be drawn, while thebitmap mask may comprise a corresponding bitmap, where each cellcorresponds to a bit. The value of the bit in the bitmap mask mayindicate whether or not the value of the overlay is to be used. As anexample, consider a frame f, an overlay bitmap image, o, and a mask, m.For each pixel (x,y), the output value (out) representing the AR finaldisplay may be computed as dis(x,y)=o(x, y)·m(x, y)+f(x, y)·(1−m(x, y)).The overlay display (od), however, may be computed asod(x,y)=o(x,y)·m(x,y). Bitwise operations may be performed on thebitmasks, such as dis=(o AND m)XOR (f AND ¬m), and od=o AND m. In someexemplary embodiments, the mask may be either 0 (do not use the overlaybit) or 1 (use the overlay bit). Additionally or alternatively, valuesin between 0 and 1 may be used to provide for opaqueness andtransparency properties.

In some exemplary embodiments, the overlay may be an image in the sizeof the frame. As the AR framework may combine the overlay with theframe, the overlay may comprise certain pixels from the output of theskin and transparent pixels when no skin output is received for acertain location in the frame. Additionally or alternatively, theoverlay may comprise an image that may be the output of the skin and itsrelative position in the frame. In some exemplary embodiments, theoverlay may comprise information regarding the importance the positionof the skin output in the frame. As an example, an informative skin thatadds labels to objects may be agnostic regarding the position of thelabel. Additionally or alternatively, a skin that adds lipstick mayconfigured to set its location requirement to a high and/or maximalscore, e.g., such as “high”, “1”, or the like.

In some exemplary embodiments, if the view of the AR device is shared,the overlay display may be overlaid over the frame to generate the ARframe, which represents the view that is perceived by the user using theAR device.

In some exemplary embodiments, Steps 230-240 may be performediteratively, while invoking different skins, until all skins that areregistered on available objects are invoked. In some exemplaryembodiments, an overlay is obtained from one skin, incorporated into theframe and potentially provided to the next skin that processes theaffected area. Additionally or alternatively, the overlays may becombined with one another, and maintained separately from the frameitself. The skins may receive the frame and the combined (previous)overlays. Additionally or alternatively, the skins may receive anaugmented frame, comprising the frame and combined (previous) overlays,for processing.

On Step 250, the final overlay display that is generated, based on thecombined overlays of the invoked skins, may be displayed. It is noted,that the overlay is seamlessly interwoven with the physical world, whichis represented by the original frame of Step 210. As a result, theoverlay display is overlaid over the physical world, without combing theframe itself.

In some exemplary embodiments, an AR framework may be operativelycoupled with one or more frame sensors. The frame sensors may produce astream of frames. The AR framework may obtain the streams. In the casethat there are several frame sensors, the AR framework may combinemultiple streams of frames into a single stream. In case that a skin isregistered on types of objects that the single stream comprises, the ARframework may apply the skin on an appropriate object for each frame. Insome exemplary embodiments, steps 210-250 may be repeated.

In some exemplary embodiments, a user may configure the AR framework toexecute a skin at a specific location, for a specific time duration, orthe like. As an example, a user may be using his AR glasses constantly.When he enters the supermarket, a skin that shows prices of products maybe added to the list of active skins and may be invoked as long as theuser is in the supermarket. When the user leaves the supermarket, theskin may be deactivated. In some exemplary embodiments, activation ordeactivation may be configured to occur when a marker is identified, asexplained hereinbelow. Additionally or alternatively, upon identifying amarker, a suggestion to activate a skin may be presented to the user andthe user may confirm or reject the suggestion. In some cases, the usermay set his personal preferences for activation or deactivation ofskins, such as based on identifying markers, based on his location,based on a context, based on a time of day, based on a socialenvironment in which he his located, or the like.

Referring now to FIG. 3 showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 310, each skin is registered on a type of an object. In someexemplary embodiments, each skin may register with the AR framework toreceive notification and handle specific types of objects. An objectappearing in the frame may be identified and classified to a type. TheAR framework may support a variety of objects types, such as but notlimited to a person, an animal, a house, a road, a plant, a tree, aface, an eye, or the like. In some exemplary embodiments, an object maybe classified as having a plurality of object types, such as objectsthat are an abstraction or instantiation of one another (e.g, a policeofficer may be an instantiation of an abstract person type), objectsthat have more than one attribute corresponding to an object type (e.g.,a person may be both an adult, a female and a Caucasian), or the like.In some exemplary embodiments, an object may comprise other objects. Asan example, an object of type person may comprise objects of type face,eyes, arms, clothes, or the like.

On Step 320, the AR framework obtains a frame. The frame may be obtainedfrom a frame sensor such as a camera that is embedded in an AR Glasses,from a camera that is embedded in a mobile phone, from a surveillancecamera, from a thermographic camera, from an infrared camera, or thelike. The frame sensor may capture one frame after the other and producea stream of frames. The frame sensor may obtain frames that may beprocessed to generate output overlays for the frames in a rate such as90 Frames per Second (FPS), 60 FPS, or the like. In some exemplaryembodiments, a rate which is lower than 60 may result in a quality whichmay be poor for human users.

The AR framework may obtain a frame from a frame sensor and may send theframe, transmit the frame, save the frame, or the like for futureprocessing. In some exemplary embodiments, the AR framework may obtainseveral frames from multiple sources in a single time unit and merge theframes to a single frame prior to processing the frame. As an example,the AR framework may not have access to a frame as viewed from the fieldof view of the user. Using a plurality of cameras, several differentimages may be transformed and merged together to provide an estimatedframe of the view of the user. The image of the room or of the personmay be taken from several image sensors. Frames from different imagesensors may be merged to a single frame.

On Step 330, the AR framework identifies objects that appear in theframe. The AR framework may employ object recognition, computer visiontechniques, Artificial Intelligent (AI) algorithms, or the like, toidentify the objects in the frame. Additionally or alternatively,metadata about objects may be available in other manners, such as usingwireless transmission, RFID, consultation with 3D maps, or the like.

In some exemplary embodiments, the AR framework may classify the objectsto types and may build a list comprising the types of the objects thatcomprised by the frame. In some exemplary embodiments, the AR frameworkmay create or maintain a metadata record. The metadata record may beupdated to comprise each identified object and properties thereof. Forexample, the metadata record may indicate for an object its type, itslocation in the frame, its dimensions, or the like. In some cases, skinsthat are executed by the AR framework may access the metadata record,utilize its content, and potentially update it.

In some exemplary embodiments, an object may not have existed in theframe as captured by the frame sensor. A skin may process the frame orportion thereof, and add an object record to the metadata record toindicate the existence of the new object, to ensure it is processedcorrectly. As an example, a skin may add a label to be presented near anexisting object, a skin may add an object of a cat, a navigation skinmay add a graphical representation of a route on top of the road, or thelike. As another example, the AR framework may add images of familymembers of a person that is shown in a frame, a garden near a house, orthe like.

On Step 340, the AR framework invokes a skin which is registered on thetype of the detected object in the frame. As each skin may haveregistered on types of objects which are of interest to it (e.g., Step310), the AR framework may invoke only skins that have objects ofinterest in the frame to avoid having each skin determine, dynamically,whether objects of interest appear in the frame, requiring resourceconsumption, such as energy and processing resources.

In some exemplary embodiments, the AR framework may be executed by adevice, such as but not limited to the AR device (e.g., AR Glasses 120)itself, an edge computer (e.g., Edge 130, 130′ of FIG. 1), a server(e.g. Server 140 of FIG. 1). In some cases, each skin may be executed ona different device, for example, on an AR device (e.g., AR Glasses 120of FIG. 1), on an edge computer (e.g., Edges 130,130′ of FIG. 1), on aserver (e.g., Server 140 of FIG. 1), or the like. The AR framework mayinvoke the skin on the relevant computing environment to process theobjects of interest. As an example, the skin may begin execution upon acommand of the AR framework. As another example, the skin may beexecuted by a daemon process that pre-exists and begins processing inresponse to a command from the AR framework. In some cases, if severalskins are registered on the same object, the AR framework may invokethem one after the other.

In some exemplary embodiments, the AR framework may provide the invokedskin with a frame, a portion of the frame (also referred to as a “subframe” or “patch”), or the like. In some exemplary embodiments, the ARframework may provide the skin with the sub frame that consists of theobject of interest. If several objects appear, the AR framework mayinvoke the skin a plurality of times, each of which with a different subframe. Additionally or alternatively, the AR framework may provide theskin with the metadata of the object, for accessing and potentially forupdating. For example, if the skin performs facial recognition, the skinmay update the metadata properties of the object to indicate theidentified person of the object. Other skins that perform facialrecognition may avoid performing facial recognition and rely on theoutput of a previous skin. Additionally or alternatively, the output ofthe previous skin may be verified prior to being relied upon.

Additionally or alternatively, the AR framework may provide the skin asan input an editable section of the frame. The editable section may bean area in the frame or patch on which the output of the skin may bedrawn.

Additionally or alternatively, the input provided to the skin maycomprise the metadata record. The metadata record may comprise data suchas a list of objects that are in the frame or patch, properties of theobjects, coordinates on which the skin is allowed to draw on, thecontext in which the frame was taken, properties regarding the personusing the AR device, or the like. In some exemplary embodiments, acontext of the frame may comprise the location in which the frame wastaken, information about the location (e.g. a private house, a workplace, information from Geographic Information System (GIS), or thelike), the date in which the frame was taken, data regarding the date(e.g. a birthday, an anniversary, or the like), data regarding the framesensor used to obtain the frame, or the like.

In some exemplary embodiments, the skin may modify the object in aconstructive manner, such as by augmenting the object with additionalobjects, decorations, ornaments, or the like. Additionally oralternatively, the skin may modify the object in a destructive manner,such as painting over the object to remove the object, replacing theobject by another object, or the like.

In some exemplary embodiments, the skin may provide an output. Theoutput may comprise an overlay to be displayed over the frame or patch,as determined by the skin. As an example, a skin may add a balloon toappear in a hand of a child. The output may thus by an overlay that,when applied over the frame, adds the balloon at the determinedlocation. As another example, a skin may add a green line on a road,indicating a route on the road. In that case, the output of the skin maybe provided using a set of coordinates and a corresponding color,defining the overlay implicitly. As yet another example, the output ofthe skin may comprise a label with text comprising information about theinput object.

Additionally, or alternatively, the skin output may comprise themetadata record. In some cases, the metadata record may be updated bythe skin to add information gathered by the skin to existing objects, toadd new objects (e.g., the balloon added in the example above), or thelike. Additionally, or alternatively, the metadata record may comprisepainting instructions to be implemented by the AR framework. As anexample, if a skin paints a label before having the final augmentedframe, labels may be positioned in a sub-optimal location, such as in adistribution that is too dense, blocking important objects or othervisual information, or the like. The skins may add instruction for theAR framework to add labels to objects, such as a name near a person, amaterial of a structure near the structure, a price tag near amerchandise, a speed indication near a windshield, or the like. Afterall the skins are executed, and after the overlay display is generated(Step 380), the AR framework may determine the location, size, and othervisual properties of the labels, and draw them, to add an additionaloverlay layer, referred to as a label overlay.

In some exemplary embodiments, the metadata record outputted by the skinmay be the updated metadata record. Additionally, or alternatively, themetadata record may comprise the added information only, so as to allowthe AR framework to modify the metadata record accordingly at itsdiscretion.

On Step 350, the AR framework obtains an overlay that is the output ofthe skin. In case that the skin is being executed in the same apparatusas the AR framework, the overlay may be pointed by a pointer that wasthe input of the skin, or the like. In some exemplary embodiments, theskin may be executed on a different execution platform, such as an edgecomputer, a server, or the like. The remote platform may send, transmit,or the like, its output as a stream to the AR framework. Additionally oralternatively, the executing platform may store the output of the skinon a local storage device, on a remote cloud server, or the like. The ARframework may access the medium on which the output of the skin isretained and pull the overlay.

In some exemplary embodiments, the skin may output the first overlayfollowed by outputting updates when its output changes. Each outputbeside the first overlay may be the delta from the previous output.

As explained above, in some cases, labels may be added at a later stage,allowing other objects to be drawn, as the location of the label may beless important compared to the location of other objects. As an example,in some cases, the label may be drawn far from the object it isassociated with and connected thereto using an arrow, a line, or thelike. As an example, a label with information about an historical placemay be above, below, on the left or on the right of the image of thatplace. The AR framework may determine the exact location of the label,its size, and other graphical properties after the overlay display isgenerated.

On Step 360, it may be determined if there is another skin that isregistered on the same type of object. If so, Steps 340 and 350 may beperformed again with the respect to the next skin, which may process thesame objects. Otherwise, Step 370 may be performed.

On Step 370, in case there is another object in the frame that was notyet processed, Steps 330-360 may be repeated with respect to theadditional object. As a result, the AR framework may iterate over eachobject that is in the frame and invoke skins thereon. Otherwise, Step380 may be performed.

It may be noted that in some cases, skins may be executed in parallel.In some cases, skins that are registered on different types of objectsmay be executed in parallel to reduce overall computation time. In somecases, a first skin may be registered on a first type of object before asecond skin. The second skin may also be registered on another type ofobject. The first skin may process an object of the first type inparallel to the second skin processing an object of the second type.After the first skin completes its processing, the second skin mayprocess the object of the first type. As another example, if there are aplurality of objects of the first type, the first skin may process thefirst object, provide an output for the second skin to process the firstobject sequentially. While the second skin processes the first object,the first skin may continue to process, in parallel, a next object ofthe same type, creating a pipeline between the first skin and the secondskin. In some exemplary embodiments, parallel execution may be performedon different execution platforms. Additionally or alternatively,parallel execution may be useful to reduce latency which may be causedby extra processing time. In some cases, the final output may beprovided within a target time, so as to enable an output display streamat a desired rate, such as for example, a rate of 20 FPS, 60 FPS, 90FPS, or the like (e.g., ±10%).

On Step 380, the AR framework may combine overlays from different skinsto provide an output overlay. The output overlay may be displayed to theuser, providing an augmented reality experience with augmentation overthe frame viewed by the user. Additionally or alternatively, a labeloverlay may be generated by the AR framework and combined with theskin-generated overlays.

In some exemplary embodiments, the AR framework may add the overlay ofeach skin to the overlay as soon as the overlay is available.Additionally or alternatively, it may be useful to postpone adding anoverlay in order to save processing time. A second skin may beregistered on an object that is the output of the first skin. The secondskin may change the objects that the first skin generated. Hence, theremay be no need to draw the output of the first skin, and any computationtime spent in adding it may be wasted.

In some exemplary embodiments, the metadata record which may be passedbetween the skins and the framework and potentially modified by theskins, may comprise privacy settings. The privacy settings may besettings of an object appearing in the frame or in the overlay display.As an example, the privacy settings may indicate whether the image maybe retained, may be shared with other people, or the like. Privacysettings may be defined by the user, may be updated by the skins, may beupdated by the AR framework in response to identification of a marker,or the like. In some cases, the AR framework may enforce therestrictions defined by the privacy settings. For example, if theprivacy settings allow retaining and sharing of the frame, the frame maybe retained and shared. Additionally or alternatively, the AR frameworkmay combine the frame and display overlay for generating the AR view tobe shared and retained for future use. If the privacy settings do notallow sharing of the frame, the frame itself may not be shared withothers. Additionally or alternatively, if the privacy settings preventthe sharing or storage of specific objects in the frame, the ARframework may edit and redact portions of the frame, of the displayoverlay, combination thereof, or the like, based on the privacysettings.

On Step 390, the overlay display is displayed using the AR device. As aresult, the overlay display is overlaid over the physical world, withoutcombining the frame itself. In some exemplary embodiments, the overlaydisplay is displayed by the AR glasses, providing personalized AR viewof the world to the user. Additionally or alternatively, the AR devicemay be the windshield of a vehicle, providing a passenger of the vehiclewith an augmented view of the surroundings of the vehicle. In otherembodiments, the AR device may be located at any other location and/ordevice.

Referring now to FIG. 4A showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 400, a privacy policy of a user may be obtained. The privacypolicy may comprise instructions instructing on which computerizedenvironment to execute the trusted task and on which computerizedenvironment to execute the non trusted tasks. Additionally oralternatively, the privacy policy may define what tasks are privatetasks, what tasks are non-private tasks, or the like. Additionally oralternatively, the privacy policy may define which devices are trusteddevices, which devices are non-trusted devices, or the like. In someexemplary embodiments, it may desired to execute private tasks ontrusted devices, non-private tasks on a non-trusted device, or the like.Additionally or alternatively, non-private tasks may be executed on anydevice that may be available in terms of computational resources. Insome exemplary embodiments, the decision of which device to utilize toexecute the non-private task may be based on an optimization of theresources available in the system, which include the trusted andnon-trusted devices. Additionally or alternatively, the privacy policymay define what data is private, and what data is non-private.Additionally, or alternatively, the privacy policy may define networksthat are private, networks that are non-private, or the like. As anexample, the privacy policy may define that facial recognition is aprivate task. As another example, the privacy policy may define that theAR device are a trusted device, that a first cloud server is a trusteddevice and that a second cloud server is a non-trusted device.Additionally, or alternatively, the operation of the skin itself, itsoutput, the functionality thereof, or the like may be the private datathat is not to be disclosed. In some cases, a skin may declare that anoutput it provided is private.

In some exemplary embodiments, a trusted device for processing a privatetask may not be available. The device may not be available due to anoverload on the device itself, due to an internet overload, or the like.The privacy policy may comprise instructions regarding the operation ofthe AR device in the absence of a trusted devices. As an example, in theabsence of a trusted device, the AR device may pause the execution ofthe private task. Additionally, or alternatively, in the absence of atrusted device, the AR device may pause its operation until a trusteddevice is available.

In some exemplary embodiments, the privacy policy may define a privacyscore that may be associated with a task. Additionally, oralternatively, the privacy policy may define a trust score that may beassociated with each device that may execute tasks. In some exemplaryembodiments, an AR device may be configured to execute a task associatedwith a privacy score on a device with a higher trust score than theprivacy score.

In some exemplary embodiments, the privacy policy may comprise a privacycontext. The privacy context may comprise data such as a location inwhich the privacy policy is valid, a duration in which the privacypolicy is valid, other people that at their presence the privacy policyshould be enforced, or the like.

In some exemplary embodiments, a user may wish to hide from a thirdparty an execution of a skin at an AR device. Hence, the privacy policymay comprise that the skin on a trusted device.

On Step 410, a frame is obtained. The frame may be obtained from atleast one frame sensor that is operationally coupled with an AR device.In some exemplary embodiments, a frame may be obtained every shortperiod of time, e.g., every 90 milliseconds, every 60 milliseconds,every 50 milliseconds, every 30 milliseconds, or the like.

On Step 420, based on the privacy policy, a private task and anon-private task may be determined. In some exemplary embodiments, theprivate task may be a task comprising processing private data.Additionally, or alternatively, the private task may comprise sendingprivate information. Additionally, or alternatively, the private taskmay comprise sending private information over a trusted network.Additionally, or alternatively, the private task may relate to an outputof a skin, such as an output that was declared to be private.

On Step 430, an overlay display for the frame may be generated. In someexemplary embodiments, an overlay display may be based on execution ofprivate and non-private tasks. Additionally or alternatively, an overlaydisplay may comprise outputs of the private task, output of thenon-private task, the original frame, a portion of the original frame,or the like. In some exemplary embodiments, the overlay display may begenerated after performing Step 433, Step 435 or any combinationthereof.

On Step 433, the non-private task may be executed by a non-trusteddevice.

On Step 435, the private task may be executed by a trusted device. Insome exemplary embodiments, the trusted device may be the AR device.Additionally, or alternatively, the AR device may be a non-trusteddevice. As an example, a user may rent an AR device, borrow an ARdevice, or the like. The owner of the AR device may be able to obtain aportion of the data from the AR device. Hence, the AR device may not beconsidered as a trusted device. In some exemplary embodiments, an edgedevice may be a trusted device. The edge device may be a personalcomputer, a smart phone of the user, or the like. Additionally oralternatively, the edge device may be a non-trusted device. As anexample, a user may rent a car. The car infotainment system may be theedge device. As the user has no control over the data that the carinfotainment system may collect, the edge device may be a non-trusteddevice.

In some exemplary embodiments, in case that the private task comprisesprivate data that should be sent to a trusted device, executing theprivate task by a trusted device may refer to sending the sensitive datato a trusted device.

In some exemplary embodiments, the non-private task may be configured todetermine a high-resolution overlay of the frame. Additionally, oralternatively, the private task may be configured to determine alow-resolution overlay of the frame.

On Step 440, the overlay display may be displayed by the AR device to auser. In some exemplary embodiments, the overlay display may comprisethe first overlay that is the output of the non-private task, the secondoverlay that is the output of the private task, or the like.

Steps 400-440 may be performed repeatedly, for a stream of frames. Insome exemplary embodiments, Step 400 may be followed by Step 420. Oncethe privacy policy is obtained and the private task and the non-privatetask have been determined, Steps 430 and 440, may be repeated for astream of frames.

It is noted that Step 435 may be executed prior to Step 433, in parallelto Step 433, or after Step 433.

Referring now to FIG. 4B showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 422 b, information about a field of view of the user may bedetermined. In some exemplary embodiments, a point of fixation of theuser may be determined. The point of fixation may be an area in theframe at which the user is looking. In some exemplary embodiments, itmay be desired to hide images and objects that the user is fixating atfrom non-trusted devices. As an example, the user may be on a date,watching a show. The user may be staring at an actress' boozem, notpaying any attention to the show. The user may wish to hide it from hisdate. In some exemplary embodiments, it may be desired to hide the pointof fixation of a user from any device in order to minimize the abilityof third parties to create a psychological profile of the user. In someexemplary embodiments, hiding the point of fixation of the user mayharden advertisers to create focused profile of the user for purposes ofadvertisement optimizations.

On Step 424 b, a private task may be determined for the point offixation portion of the frame. Determining the private task may comprisedetermining a trusted device on which the private portion of the framemay be processed.

On Step 426 b, a non-private task for the remaining portion may bedetermined. The remaining portion may include areas in the frame thatare not the private portion. Determining the non-private task maycomprise determining a non-trusted device by which the non-privateportion may be processed.

Referring now to FIG. 4C showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 422 c, private objects in the frame may be identified. In someexemplary embodiments, private objects may include objects that the userwould prefer not to reveal to a third party. The user may not want thatthe third party will know with whom he is talking, his location, at whatobjects he is looking, or the like. As an example, the private objectsmay be faces of people, intimate body parts, a document having a“confidential” title, a screen showing source code of a program, or thelike.

It is noted that in some exemplary embodiments, a skin that a user isusing may be considered as private data. As an example, a user may payfor a skin that undresses people. Although a frame may not compriseprivate data the user may wish that the skin will be executed on atrusted device in order to not reveal that he is using the skin. Hence,executing the skin may be a private task.

In some exemplary embodiments, a user may wish to hide the fact that sheutilizes a certain skin. The skin may be considered as a privateinformation and the execution thereof may be a private task, to avoidrevealing the user's usage of the private skin. As an example, the skinmay be a dating skin, a skin that performs unethical operations, such asundressing bystanders, a skin that accesses non-legitimate data sources,such as dark web or illegal databases, or the like.

In some exemplary embodiments, the fact that the user utilizes a skinmay be, generally, disclosable information. However, the specificconfiguration of the skin may be considered as private information. Forexample, the user may provide parameters to a skin that affect itsoperation. The user may wish to conceal the parameters. Additionally oralternatively, the user may wish to conceal some operation of the skin,such as when operating on a specific object, a specific type of object,or the like. Accordingly, execution of the skin may be considered asprivate tasks depending on the operation of the skin, on whether or notthe parameters are provided to the execution environment, on whether thecontent of the parameters are reflected by the objects being processed,or the like. As an example, the user may configure a skin to operate onboth minors and adults, and may wish to conceal the fact that the skinis being applied on minors. Hence, execution of the skin on an objectrepresenting an adult may be considered a non-private task, whileexecution of the skin on an object representing a minor may beconsidered as a private task. Additionally or alternatively, theapplication of a skin on a minor may be considered as a private task toavoid divulging information regarding minors to untrusted parties, suchas information reflected from the frame or obtained by the skin whenoperating. As another example, the user may utilize a blocking skin thatremoves or otherwise redacts an image of a person from a frame. However,the user may wish to conceal from third parties the identity of thepeople being removed, e.g., her ex-boyfriend, a stalker, or the like.

In some exemplary embodiments, the AR framework may be configured toquery a skin whether the skin is a private task or a non-private task.Additionally, or alternatively, as the privateness of the skins maychange based on a frame, the framework may query the skin every second,every two seconds, every five seconds, or the like. Additionally, oralternatively, the framework may track objects and in response toidentifying that there is a new object in the frame that the skin isregistered on, the framework may query the skin whether executing theskin on that object is private task or non-private task.

On Step 424 c, a private task for the private objects may be determined.The private task may comprise processing the private objects on atrusted device, sending the private object over a trusted network, orthe like. As an example, a skin that is configured to perform facialrecognition may be considered a private task in order to avoid sendingfaces of people to a non-trusted device. As another example, the privatetask may be to blur the confidential document.

On Step 426 c, a non-private task for the reminder portion of the framemay be determined. The reminder portion may be areas in the frame thatare private, that comprise private objects, or the like. The non-privatetask may comprise processing the non-private data on any device that isavailable, sending the data over any network, or the like.

Referring now to FIG. 4D showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 422 d a private portion of the frame may be determined. In someexemplary embodiments, the private portion of the frame may bedetermined based on the content of the frame, such as comprisingconfidential objects, based on information embodied therein, or thelike.

On Step 431, a non-private portion of the frame may be sent to anon-trusted device. The non-private portion may be the frame without theprivate portion. The non-trusted device may be an edge computer, a cloudserver, or the like.

On Step 435, the non-private task may be executed by the non-privatedevice in order to determine a first overlay. The first overlay may bethe output of the non-private task, such as the output of a skin.

On Step 435, the private task may be executed by a trusted device inorder to determine the second overlay. In some exemplary embodiments,the AR device may be the trusted device. As the private task maycomprise processing a portion of the frame comprising an object that theuser is looking at, it may be desired to avoid sending that portion to anon-trusted device.

It is noted that the non-private task and the private task may compriseinvoking the same skin on different portions of the frame.

On Step 439, the first and second overlays may be merged after beingreceived from the trusted device and non-private device, respectively.

Referring now to FIG. 4E, showing a flowchart diagram of a method, inaccordance with some exemplary embodiments of the disclosed subjectmatter.

On Step 422 e, a peripheral portion of the frame may be determined. Insome exemplary embodiments, the peripheral portion of the frame may aportion that is sensed by the user's peripheral vision. The frame may bedivided to peripheral portion and non-peripheral portion. In someexemplary embodiments, the non-peripheral portion of the frame may be anarea in the frame corresponding to the area on which the user islooking. Identifying a direction to where the user is looking within theframe may be performed using a camera coupled with the AR device, an IRcamera coupled with the AR device, or the like. In some exemplaryembodiments, the non-peripheral portion of the frame may be a privateportion of the frame. As an example, a user may wish that an ARframework will process areas on which he as looking as a private portionof the frame. In some exemplary embodiments, processing areas of theframe as private tasks may comprise not sending the private areas to anon-trusted device, sending the private areas over a trusted network, orthe like. In some cases, peripheral portions of frames may be processeddifferently, such as in a lower resolution to conserve resources.

On Step 431 e, the peripheral portion of the frame may be sent to anon-trusted device. Step 431 e may be performed similarly to Step 431 ofFIG. 4A.

On Step 437, the resolution of the peripheral portion of the frame maybe reduced, yielding power saving while not compromising the privacy ofthe user. Additionally or alternatively, the resolution of the firstoverlay may be reduced. Additionally or alternatively, a user may beindifferent to reducing the resolution of the peripheral areas of theframe due to the peripheral areas being sensed by the user's peripheralvision.

Referring now to FIG. 5 showing an apparatus, in accordance with someexemplary embodiments of the disclosed subject matter.

In some exemplary embodiments, Apparatus 500 may comprise Processor 502.Processor 502 may be a Central Processing Unit (CPU), a microprocessor,an electronic circuit, an Integrated Circuit (IC) or the like.Additionally or alternatively, Apparatus 500 can be implemented asfirmware written for or ported to a specific processor such as DigitalSignal Processor (DSP) or microcontrollers, or can be implemented ashardware or configurable hardware such as Field Programmable Gate Array(FPGA) or Application Specific Integrated Circuit (ASIC). Processor 502may be utilized to perform computations required by Apparatus 500 or anyof its subcomponents.

In some exemplary embodiments, Processor 502 may be configured todetermine a trust score for an external computerized environment. Theexternal computerized environment may be an edge device, a cloud server,or the like. In some exemplary embodiments, a privacy score may beassociated with a frame. Processor 502 may be configured to process aframe on an external device that has a trust score that is equal to orgreater that the privacy score of the frame. Additionally, oralternatively, Processor 502 may be configured to execute a private taskon an external device that a trust score above a predetermined value.Additionally, or alternatively, a task may have a privacy score.Processor 502 may be configured to execute a task on an external deviceif the trust score of the device is greater than or equal to the privacyscore of the task.

In some exemplary embodiments, based on the trust score and on theprivacy policy, Processor 502 may be configured to determine a first setof skins as private skins. The private skins may be applied on frames bya first external device having a trust score above a threshold.Additionally, or alternatively, Processor 502 may be configured todetermine a second skins as non-private skins. The non-private skins maybe applied on frames by a second external device having a trust scorethat is below the threshold.

In some exemplary embodiments, Apparatus 500 may comprise a Display 504.Display 504 may display augmented images to a user. The augmented imagesmay be the result of frames captured by Frame Sensor 508, 509 andprocessed by the apparatus.

In some exemplary embodiments, Apparatus 500 may comprise a Frame Sensor508. In some exemplary embodiments, Frame Sensor 508 may be an integralpart of an augmented reality glasses worn by the use.

In some exemplary embodiments, Apparatus 500 may comprise a Frame Sensor509. Frame Sensor 509 may be external to Apparatus 500.

In some exemplary embodiments, Apparatus 500 may comprise an I/O Module506. Apparatus 500 may utilize I/O Module 506 as an interface totransmit and/or receive information and instructions between Apparatus500 and external I/O devices such as, Trusted Device 560, Non-TrustedDevice 570, Frame Sensors 509, or the like. Additionally, oralternatively, I/O Module 506 may be used by Apparatus 500 in order tosend frames to Trusted Device 560, to Non-Trusted Device 570, or thelike. Additionally, or alternatively, I/O Module 506 may be utilized inorder to obtain overlays from Trusted Device 560, from Non-TrustedDevice 570, or the like.

In some exemplary embodiments, Trusted Device 560 may be connected toApparatus 500. Apparatus 500 may utilize Trusted Device 560 in order toinvoke skins thereon, in order to execute a private task thereby, or thelike. Executing a private task on Trusted Device 560 may yield powersaving compared to executing the private task locally, on Apparatus 500.

In some exemplary embodiments, Non-Trusted Device 570 may be a remotecloud server. I/O Module 506 may be connected to Non-Trusted Device 570via a Wide Area Network (WAN) such as the Internet. Additionally, oralternatively, I/O Module 506 may connect with Non-Trusted Device 570via Edge 560. Apparatus 500 may utilize Non-Trusted Device 570 in orderto invoke skins thereon.

In some exemplary embodiments, Apparatus 500 may comprise a Memory Unit507. Memory Unit 507 may be persistent or volatile. For example, MemoryUnit 507 can be a Flash disk, a Random Access Memory (RAM), a memorychip, an optical storage device such as a CD, a DVD, or a laser disk; amagnetic storage device such as a tape, a hard disk, Storage AreaNetwork (SAN), a Network Attached Storage (NAS), or the like; asemiconductor storage device such as Flash device, memory stick, or thelike. In some exemplary embodiments, Memory Unit 507 may retain programcode to activate Processor 502 to perform acts associated with any ofthe steps shown in FIG. 4A-4E. In some exemplary embodiments, MemoryUnit 507 may be configured to retain a privacy policy of a user of theapparatus.

In some exemplary embodiments, Skins 550 may be executed locally inMemory Unit 507 or remotely on a computerized environment such asTrusted Device 560, Non-Trusted device 570, or the like. In someexemplary embodiments, Apparatus 500 may be configured to decide whethera skin is to be executed on a non-trusted device or on a trusted devicebased on a privacy policy. As an example, a user may be in a privatemeeting, not wishing that data regarding the meeting is processedremotely.

The components detailed below may be implemented as one or more sets ofinterrelated computer instructions, executed for example by Processor502 or by another processor. The components may be arranged as one ormore executable files, dynamic libraries, static libraries, methods,functions, services, or the like, programmed in any programming languageand under any computing environment.

In some exemplary embodiments, Apparatus 500 may comprise a PrivacyPolicy Obtainer 510. Privacy Policy Obtainer may obtain a privacy policyfrom a remote cloud server. Apparatus 500 may be configured to decidewhere to execute Skins 550, Private Skins 580, non-private Skins 590based on the privacy policy.

In some exemplary embodiments, Apparatus 500 may comprise a FrameAnalyzer 520. Frame Analyzer 520 may be configured to analyze frames inorder to determine a private portion of the frame. Additionally, oralternatively, Frame Analyzer 520 may be configured to determine aprivate object therein. A private portion of the frame, based on theprivacy policy, may be a portion displaying a face of a person, intimatebody parts, private or confidential objects such as documents, aperson's belongings, or the like. Additionally, or alternatively, FrameAnalyzer 520 may be configured to analyze frames in order to determine apoint of fixation of the user.

In some exemplary embodiments, Apparatus 500 may comprise a TasksManager 530. Tasks Manager 530 may be configured generate a privatetask, to generate a non-private task, or the like. Additionally, oralternatively, Tasks Manager 530 may be configured to determine aprivate skin from Skins 550 and to execute the skin by Trusted Device560. Additionally, or alternatively, Tasks Manager may be configured todetermine a non-private skin from Skins 550 and execute it byNon-Trusted Device 570. In some exemplary embodiments, Trusted Device560 may hold and execute Private Skins 580. Additionally, oralternatively, Non Trusted 570 may hold and execute Non Private Skins590. Private Skins 580 and Non-Private Skins 590 may run regardless withthe operation of Apparatus 500. Executing the private task may comprisesending private data to the private skin and obtaining a second overlaytherefrom. The private data may comprise a private portion of the frame,a private object, or the like. Additionally, or alternatively, executingthe non-private task may comprise sending a non-private portion of theframe to Non Private Device 590 and obtaining the first overlay.

In some exemplary embodiments, Apparatus 500 may comprise a DisplayManager 540. Display Manager 540 may be configured to obtain a firstoverlay. Additionally, or alternatively, Display Manager may beconfigured to obtain a second overlay. Display manager may be configuredto merge the first and the second overlay to form the overlay displayand to provide the overlay display to Display 504. Additionally, oralternatively, Display Manager may be configured to provide the firstoverlay, the second overlay, or the like to Display 504. In someexemplary embodiments, Display Manager 540 may be configured to reducethe resolution of the first overlay prior to providing the first overlayto Display 504, prior to forming the overlay display, or the like.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method comprising: obtaining a privacy policyof a user, wherein the user is using an Augmented Reality (AR) device;obtaining a frame from at least one frame sensor; generating an overlaydisplay for the frame, wherein said generating is based on an executionof a non-private task on a non-trusted device and based on an executionof a private task on a trusted device, wherein the private task and thenon-private task are determined based on the privacy policy; anddisplaying, by the AR device, the overlay display.
 2. The method ofclaim 1, further comprises: determining a private portion of the frame,whereby dividing the frame to the private portion and a non-privateportion; wherein the execution of the non-private task comprisesprocessing the non-private portion of the frame, whereby obtaining afirst overlay; wherein the execution of the private task comprisesprocessing the private portion of the frame, whereby obtaining a secondoverlay; wherein said generating the overlay display further comprises:merging the first overlay and the second overlay to obtain the overlydisplay.
 3. The method of claim 2, wherein said determining the privateportion of the frame is based on a point of fixation of the user,wherein the non-trusted device is unable to ascertain the point offixation of the user from the non-private portion.
 4. The method ofclaim 2, wherein said determining the private portion of the frame isbased on an identification of Personally Identifiable Information (PII)appearing in the frame.
 5. The method of claim 4, wherein the PIIcomprises visible facial features of a person, whereby the non-privateportion excludes the visible facial features.
 6. The method of claim 1further comprises determining a peripheral portion of the frame based ona point of fixation of the user; wherein the non-private task isconfigured to determine a high-resolution overlay of the frame; whereinthe private task is configured to obtain a low-resolution overlay of theperipheral portion of the frame; and wherein the overlay display for theframe comprises the low-resolution overlay of the peripheral portion ofthe frame and at least a portion of the high-resolution overlay of theframe.
 7. The method of claim 1, wherein the trusted device comprisesthe AR device, wherein the non-trusted device comprises an externaldevice, external to the AR device.
 8. The method of claim 1, wherein thetrusted device comprises a trusted server, wherein the non-trusteddevice comprises a non-trusted server.
 9. The method of claim 1, whereinthe non-trusted device comprises the AR device.
 10. The method of claim1, wherein the execution of the private task comprises: analyzing theframe to determine a private portion of the frame and a non-privateportion of the frame; and providing the non-private portion of the frameto the non-trusted device, whereby the non-trusted device performing theexecution of the non-private task with respect to the non-privateportion of the frame.
 11. The method of claim 1, wherein the executionof the private task further comprises: obtaining a first overlay fromthe non-trusted device, wherein the first overlay is generated by theexecution of the non-private task; executing one or more skins on theprivate portion of the frame, whereby generating a second overlay; andmerging the first overlay with the second overlay, whereby generatingthe overlay display.
 12. The method of claim 11, wherein the overlaydisplay is determined by applying skins on the frame or a portionthereof; based on the privacy policy, identifying, in the skins, one ormore private skins and one or more non-private skins; wherein theexecution of the non-private task comprises applying the one or morenon-private skins, whereby obtaining a first overlay; and wherein theexecution of the private task comprises applying the one or more privateskins, whereby obtaining a second overlay; wherein said generatingfurther comprises combining the first overlay and the second overlay,whereby generating the overlay display.
 13. The method of claim 12,wherein the execution of the non-private task further comprises applyingthe one or more redundant skins, wherein the private task comprisesignoring output of the one or more redundant skins.
 14. The method ofclaim 12, wherein said applying the one or more private skins isperformed on a first object in the frame, wherein the execution of thenon-private task comprises applying the one or more private skins on asecond object in the frame.
 15. An apparatus comprising: a processor anda coupled memory, wherein the processor is configured to obtain aprivacy policy of a user, wherein the user is using an Augmented Reality(AR) device; obtain a frame from at least one frame sensor; generate anoverlay display for the frame, wherein the overlay display is generatedbased on an execution of a non-private task on a non-trusted device andbased on an execution of a private task on a trusted device, wherein theprivate task and the non-private task are determined based on theprivacy policy; and display, by the AR device, the overlay display. 16.The apparatus of claim 15, wherein the apparatus comprises the ARdevice.
 17. The apparatus of claim 15, wherein the processor is furtherconfigured to: determine a private portion of the frame, wherebydividing the frame to the private portion and a non-private portion;wherein the execution of the non-private task comprises processing thenon-private portion of the frame, whereby obtaining a first overlay;wherein the execution of the private task comprises processing theprivate portion of the frame, whereby obtaining a second overlay;wherein the overlay display is generated by merging the first overlayand the second overlay.
 18. The apparatus of claim 15, wherein theexecution of the private task comprises: analyzing the frame todetermine a private portion of the frame and a non-private portion ofthe frame; and providing the non-private portion of the frame to thenon-trusted device, whereby the non-trusted device performing theexecution of the non-private task with respect to the non-privateportion of the frame.
 19. A computer program product comprisingnon-transitory computer readable medium retaining instructions, whichinstructions when executed by a processor, cause the processor to:obtain a privacy policy of a user, wherein the user is using anAugmented Reality (AR) device; obtain a frame from at least one framesensor; generate an overlay display for the frame, wherein thegeneration of the overlay display is based on an execution of anon-private task on a non-trusted device and based on an execution of aprivate task on a trusted device, wherein the private task and thenon-private task are determined based on the privacy policy; anddisplay, by the AR device, the overlay display.